KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 by dejan2609 · Pull Request #20866 · apache/kafka

dejan2609 · 2025-11-12T13:31:27Z

Prologue:
#19513 (comment)

💡 Rationale: Gradle 10 will be released in Q4 2026 or Q1 2027
and we can't expect that Shadow plugin version 8 will be compatible:

Release notes for Shadow plugin:

https://gradleup.com/shadow/changes/#931-2026-01-06

I did some testing, and results look just fine ✅:

gradle builds ./gradlew :jmh-benchmarks:shadowJar and ./gradlew releaseTarGz are successful
generated distributions (kafka_2.13-4.3.0-SNAPSHOT-site-docs.tgz and
and kafka_2.13-4.3.0-SNAPSHOT.tgz) have identical file numbers (for
trunk and this branch)
Kafka cluster starts and messages can be produced and consumed:
https://kafka.apache.org/quickstart

⚠️ ~~disclaimer notes:~~

~~solution is generated via Gemini CLI~~
~~Gemini prompt is based on~~ Jira ticket (previously created by me):
KAFKA-19761

dejan2609 · 2025-11-12T20:24:06Z

@chia7712 Please review when you have some time.

build.gradle

dejan2609 · 2025-11-20T16:21:22Z

I'll rebase this onto trunk (to include a new, updated Gradle version into this branch/PR).

dejan2609 · 2025-12-06T15:07:08Z

Hi @apoorvmittal10,

Can you please allocate some time to provide a review ?

dejan2609 · 2025-12-13T15:17:45Z

Few tests are flaky/failing:

dejan2609 · 2025-12-13T15:18:08Z

Gradle task that ends up with one failed test works as expected on my machine:

dejan@dejan-HP-ProBook-450-G7:~/kafka$ date
Sat 13 Dec 16:21:53 CET 2025
dejan@dejan-HP-ProBook-450-G7:~/kafka$ java -version 
openjdk version "17.0.17" 2025-10-21
OpenJDK Runtime Environment Temurin-17.0.17+10 (build 17.0.17+10)
OpenJDK 64-Bit Server VM Temurin-17.0.17+10 (build 17.0.17+10, mixed mode, sharing)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ ./gradlew :streams:upgrade-system-tests-24:test -q
Starting build with version 4.3.0-SNAPSHOT (commit id 4196380b) using Gradle 9.2.1, Java 17 and Scala 2.13.18
Build properties: ignoreFailures=false, maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
dejan@dejan-HP-ProBook-450-G7:~/kafka$ date
Sat 13 Dec 16:22:18 CET 2025
dejan@dejan-HP-ProBook-450-G7:~/kafka$

💡 I will try to execute entire test suite on Github Action CI server once more.

dejan2609 · 2025-12-14T10:32:08Z

All tests are fine now ✅

@chia7712 can you allocate some time to review this one ?

clolov · 2026-01-09T14:39:16Z

Thanks for the change and apologies for the delay!

It may be because my expertise in Gradle is not my strong suit, but I would encourage you to give a brief overview of why all the changes in the PR are required!

The real problem is that if we just bump Gradle Shadow from 8 to 9 we run into

* What went wrong:
Execution failed for task ':jmh-benchmarks:shadowJar'.
> Cannot expand ZIP '/Users/REDACTED/Documents/kafka/core/build/classes/java/test' as it does not exist.

which appears to be because :core only has build/classes/scala while all other dependencies have build/classes/java and Gradle Shadow 9 appears to be more strict in requiring that directory to exist.

I am a bit on the fence for this change. On one hand, I don't like depending on .sourceSets.test.output in some places in the build.gradle file and on configurations/artifacts in others. On the other hand, I also don't like generating an empty folder (i.e. build/classes/java) just to satisfy a stricter expectation.

What are the opinions of others in the community?

dejan2609 · 2026-01-09T15:47:37Z

Hi @clolov and thanks for a review/comment ! You are rasing some valid questions here.

But first: can you please share your environment details (thing is that Gradle build task :jmh-benchmarks:shadowJar works just fine on my end):

dejan@box:~/kafka$ date
Fri  9 Jan 16:24:51 CET 2026
dejan@box:~/kafka$ uname -r
6.8.0-90-generic
dejan@box:~/kafka$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.3 LTS
Release:	24.04
Codename:	noble
dejan@box:~/kafka$ java -version 
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment Temurin-25.0.1+8 (build 25.0.1+8-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.1+8 (build 25.0.1+8-LTS, mixed mode, sharing)
dejan@box:~/kafka$ git log -2 --oneline 
70dac31e9f (HEAD -> KAFKA-19761, origin/KAFKA-19761) KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1
2f90af70ee (upstream/trunk, upstream/HEAD, origin/trunk, origin/HEAD, trunk) MINOR: Remove onPartitionsDeleted from GroupCoordinator interface (#21263)
dejan@box:~/kafka$ ./gradlew clean :jmh-benchmarks:shadowJar -q
Starting build with version 4.3.0-SNAPSHOT (commit id 70dac31e) using Gradle 9.2.1, Java 25 and Scala 2.13.18
Build properties: ignoreFailures=false, maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
MessageGenerator: processed 4 Kafka message JSON file(s).
MessageGenerator: processed 2 Kafka message JSON file(s).
MessageGenerator: processed 26 Kafka message JSON file(s).
MessageGenerator: processed 44 Kafka message JSON file(s).
MessageGenerator: processed 197 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::arrayBaseOffset has been called by net.openhft.hashing.UnsafeAccess (file:/home/dejan/.gradle/caches/modules-2/files-2.1/net.openhft/zero-allocation-hashing/0.16/ca252f328160ed5d027f100a4fe525d6d21daaf/zero-allocation-hashing-0.16.jar)
WARNING: Please consider reporting this to the maintainers of class net.openhft.hashing.UnsafeAccess
WARNING: sun.misc.Unsafe::arrayBaseOffset will be removed in a future release
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/dejan/kafka/coordinator-common/src/test/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntimeTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: /home/dejan/kafka/group-coordinator/src/test/java/org/apache/kafka/coordinator/group/streams/StreamsGroupTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
dejan@box:~/kafka$ sdk use java 17.0.17-tem

Using java version 17.0.17-tem in this shell.
dejan@box:~/kafka$ java -version
openjdk version "17.0.17" 2025-10-21
OpenJDK Runtime Environment Temurin-17.0.17+10 (build 17.0.17+10)
OpenJDK 64-Bit Server VM Temurin-17.0.17+10 (build 17.0.17+10, mixed mode, sharing)
dejan@box:~/kafka$ ./gradlew clean :jmh-benchmarks:shadowJar -q
Starting build with version 4.3.0-SNAPSHOT (commit id 70dac31e) using Gradle 9.2.1, Java 17 and Scala 2.13.18
Build properties: ignoreFailures=false, maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
MessageGenerator: processed 4 Kafka message JSON file(s).
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
MessageGenerator: processed 2 Kafka message JSON file(s).
MessageGenerator: processed 1 Kafka message JSON file(s).
MessageGenerator: processed 26 Kafka message JSON file(s).
MessageGenerator: processed 44 Kafka message JSON file(s).
MessageGenerator: processed 197 Kafka message JSON file(s).
MessageGenerator: processed 5 Kafka message JSON file(s).
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/dejan/kafka/coordinator-common/src/test/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntimeTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: /home/dejan/kafka/group-coordinator/src/test/java/org/apache/kafka/coordinator/group/streams/StreamsGroupTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
dejan@box:~/kafka$ date
Fri  9 Jan 16:38:25 CET 2026
dejan@box:~/kafka$

dejan2609 · 2026-01-09T16:07:43Z

I am a bit on the fence for this change. On one hand, I don't like depending on .sourceSets.test.output in some places in the build.gradle file and on configurations/artifacts in others. On the other hand, I also don't like generating an empty folder (i.e. build/classes/java) just to satisfy a stricter expectation.

I can relate with all of this @clolov. Thing is that Kafka Gradle build is complicated due to a fact that people want Kafka to perform various complicated things.

And on top of that: dependencies versions just have to move forward: sooner or later this Shadow gradle plugin version will have to be bumped, one way or another.

All-in-all, we can either choose to:

put some duct tape and/or WD-40 and call it a day (like here, in this PR 🙂)
re-factor Gradle build (and remove some technical debt all along) - ⚠️ pitfall - it can be tricky
do something else 🤓

Open invitation to entire community: what is your take on this ?

dejan2609 · 2026-01-09T18:08:04Z

It may be because my expertise in Gradle is not my strong suit, but I would encourage you to give a brief overview of why all the changes in the PR are required!

Rationale section is added into description above.

chia7712 · 2026-01-10T17:47:46Z

What are the opinions of others in the community?

The issue stems from implementation project(':core').sourceSets.test.output including non-existent paths due to joint compilation. I actually reported this Gradle before (see gradle/gradle#35293).

Instead of working around it, I think it's better to refactor the JMH module and remove the dependency on core test output. This saves us the trouble of dealing with missing paths. I have opened https://issues.apache.org/jira/browse/KAFKA-20060 for this

dejan2609 · 2026-01-11T16:39:54Z

Instead of working around it, I think it's better to refactor the JMH module and remove the dependency on core test output.

I concur with that @chia7712.

Moving this PR into draft state.

chia7712 · 2026-01-22T13:57:59Z

@dejan2609 #21310 is already merged, so please rebase code. thanks!

dejan2609 · 2026-01-22T19:37:05Z

Done, PR is rebased. I did some smoke testing (https://kafka.apache.org/quickstart) and everything seems fine (messages can be produced and consumed).

Notes/remarks:

given the fact that this is now a simple one-line change we could opt to also add Gradle minor version update here (from 9.2.1 to 9.3.0)
⚠️ pitfall: Gradle 9.3.0 has some known issues (https://docs.gradle.org/9.3.0/release-notes.html#known-issues) so maybe we should stay on the safe side and wait for Gradle 9.3.1 or 9.4.0

@chia7712 let me know what do you think about this.

release notes: https://gradleup.com/shadow/changes/#931-2026-01-06

release notes: - https://docs.gradle.org/9.3.0/release-notes.html - https://docs.gradle.org/9.3.1/release-notes.html ℹ️ Notes: - done by-the-book (procedure mentioned here `gradle/wrapper/README.md` was followed) - it makes sense to merge this with Gradle Shadow plugin version upgrade: #20866 Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

github-actions bot added triage PRs from the community build Gradle build or GitHub Actions small Small PRs labels Nov 12, 2025

dejan2609 commented Nov 12, 2025

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

github-actions bot removed the triage PRs from the community label Nov 13, 2025

dejan2609 force-pushed the KAFKA-19761 branch 2 times, most recently from e96bddd to 1292e30 Compare November 20, 2025 11:08

apoorvmittal10 added the ci-approved label Nov 20, 2025

dejan2609 force-pushed the KAFKA-19761 branch from 1292e30 to 4543ec2 Compare November 20, 2025 16:21

dejan2609 force-pushed the KAFKA-19761 branch 2 times, most recently from fcaf575 to fbb1dc5 Compare November 30, 2025 21:38

dejan2609 force-pushed the KAFKA-19761 branch from fbb1dc5 to 1354d5d Compare December 6, 2025 11:19

dejan2609 changed the title ~~KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.2.2~~ KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.0 Dec 6, 2025

dejan2609 force-pushed the KAFKA-19761 branch from 1354d5d to 19bc87c Compare December 13, 2025 08:26

dejan2609 force-pushed the KAFKA-19761 branch from 19bc87c to 4196380 Compare December 13, 2025 15:18

dejan2609 changed the title ~~KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.0~~ KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1 Jan 8, 2026

dejan2609 force-pushed the KAFKA-19761 branch from 4196380 to 70dac31 Compare January 8, 2026 23:59

dejan2609 marked this pull request as draft January 11, 2026 16:40

dejan2609 force-pushed the KAFKA-19761 branch from 70dac31 to 9267ac9 Compare January 22, 2026 19:22

dejan2609 marked this pull request as ready for review January 22, 2026 19:23

dejan2609 mentioned this pull request Jan 29, 2026

MINOR: Gradle version update: 9.2.1 -->> 9.3.1 #21372

Merged

KAFKA-19761 Gradle Shadow plugin version upgrade: 8.3.9 -->> 9.3.1

dd57627

release notes: https://gradleup.com/shadow/changes/#931-2026-01-06

dejan2609 force-pushed the KAFKA-19761 branch from 9267ac9 to dd57627 Compare February 8, 2026 08:42

Conversation

dejan2609 commented Nov 12, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented Nov 12, 2025

Uh oh!

Uh oh!

dejan2609 commented Nov 20, 2025

Uh oh!

dejan2609 commented Dec 6, 2025

Uh oh!

dejan2609 commented Dec 13, 2025

Uh oh!

dejan2609 commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented Dec 14, 2025

Uh oh!

clolov commented Jan 9, 2026

Uh oh!

dejan2609 commented Jan 9, 2026

Uh oh!

dejan2609 commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented Jan 9, 2026

Uh oh!

chia7712 commented Jan 10, 2026

Uh oh!

dejan2609 commented Jan 11, 2026

Uh oh!

chia7712 commented Jan 22, 2026

Uh oh!

dejan2609 commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dejan2609 commented Nov 12, 2025 •

edited by github-actions bot

Loading

dejan2609 commented Dec 13, 2025 •

edited

Loading

dejan2609 commented Jan 9, 2026 •

edited

Loading

dejan2609 commented Jan 22, 2026 •

edited

Loading